To ensure that the grouped-GEMM MoE loader wrapper is always active and cannot be bypassed, it is highly recommended to apply this patch before calling PatchFastRL(FastLanguageModel = FastLlamaModel). If PatchFastRL or any other downstream registration copies or wraps from_pretrained or get_peft_model from FastLlamaModel, applying this patch afterwards might result in the RL-patched entry points using the unwrapped/original methods, silently missing the MoE optimization.

The grouped-MoE loader wrappers are installed before PatchFastRL(FastLanguageModel = FastLlamaModel), so any downstream RL patching wraps the already-wrapped from_pretrained / get_peft_model.

Enable grouped MoE only after PEFT adapters attach

When FastLanguageModel.from_pretrained loads a PEFT adapter repo, loader.py calls this leaf loader first (dispatch_model.from_pretrained around line 788) and only attaches the adapter later with PeftModel.from_pretrained around lines 865-878. Wrapping the leaf here enables grouped MoE on the base model before any LoRA modules exist, so the eligibility check cannot reject LoRA-on-expert adapters and there is no later recheck; adapters targeting MoE expert projections such as the default gate/up/down targets then keep the grouped forward built from base weights and silently ignore those LoRA weights. Move the enable step to after PEFT attachment or rerun/restore the grouped patch after adapters are loaded.

Useful? React with 👍 / 👎.

The FastLanguageModel adapter-load path already re-runs auto_enable_grouped_moe after patch_peft_model in loader.py, so a loaded expert-LoRA adapter falls back to the original loop once attached. Mirrored the same re-eval onto the vision path in 25a03e2.

Move grouped MoE enable after adapter attachment

When FastModel/FastVisionModel loads a PEFT adapter repo (is_peft), loader.py first calls FastBaseModel.from_pretrained and only attaches the adapter later via PeftModel.from_pretrained. Wrapping this leaf enables grouped MoE on the base model before any LoRA modules exist, so the no-LoRA-on-experts eligibility check can pass and there is no later recheck after adapter attachment; expert-targeted MoE adapters can then run through the grouped forward built from base weights and silently ignore those adapter weights. Please move or rerun the grouped-MoE hook after the is_peft adapter attach path as well.

Useful? React with 👍 / 👎.

Fixed in 25a03e2. The FastModel / FastVisionModel is_peft path now re-runs auto_enable_grouped_moe after PeftModel.from_pretrained, so an expert-LoRA block restores the original loop while an attention-only adapter keeps the grouped path, matching the FastLanguageModel loader path.

Point the grouped-MoE import at a shipped module

In normal installs using the declared unsloth_zoo>=2026.6.7 dependency, this module path does not resolve, and the broad except silently skips the whole wrapping block. That means transformers<5 ModuleList MoE models loaded through this path (and the matching vision.py block) keep using the old expert-loop implementation instead of the intended grouped-GEMM patch, so the new auto-enable behavior is effectively a no-op unless users happen to have an unpublished zoo build.

Useful? React with 👍 / 👎.

Intentional. The import is a guarded no-op until unsloth-zoo#837 lands, which ships this exact module (temporary_patches/moe_grouped_modulelist.py). This loader change is meant to land independently and activate once the zoo side merges, so the path already points at the shipped module.

Patch PEFT adapter loads after adapters attach

When a user loads an existing LoRA/PEFT adapter repo through FastLanguageModel.from_pretrained, this wrapped leaf only returns the base model; loader.py attaches the adapter afterward via PeftModel.from_pretrained and dispatch_model.patch_peft_model at lines 865-878. The grouped-MoE wrapper therefore never sees the final PEFT model for that common adapter-loading path, while wrapping get_peft_model only covers newly-created adapters, so loaded MoE adapter checkpoints miss the intended PEFT-aware grouped-GEMM conversion.

Useful? React with 👍 / 👎.

Covered by the post-attach auto_enable_grouped_moe re-eval in the FastLanguageModel loader path, which runs after PeftModel.from_pretrained on the loaded-adapter path. Same re-eval now added to the vision path in 25a03e2.